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Web page. As results of the evaluations, lesson maps, examples of the 
calculations, glossaries, search, and more multimedia presentations have been 
added to the courses. (Author/MES) 
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Abstract: The purpose of this paper is to report the various formative evaluation activities that have been 
conducted for the Masters of Science in Agronomy Distance Educadon Program at Iowa State University and to 
provide useful recommendations for educators involved in Web-based course development It was found that 
those faculty members having experiences with distance education held more positive perceptions toward Web- 
based instruction Students believed they were learning as much in the distant environment as they would on 
campus. They liked the interactive multimedia activities and asked for clear structure for navigation in each 
web page. As results of the evaluations, lesson mcq)s, examples of the calculations, glossaries, search, and 
more multimedia presentations have been added to the courses. 



Introduction 

Distance education has a long history of applying technologies in delivering instruction to those 
who are not able to participate in a campus-b^ed curriculum. Many people involved in distance education 
are professional individuals who are seeking to pursue advanced degrees and empower themselves with 
knowledge that is relevant to their career field (Butler, 1996). The latest in the long line of instructional 
technologies is the World Wide Web (WWW). The World Lecture Hall listed almost 800 Web-based 
courses that were delivered by higher educational institutions, and this list had been growing daily (W orld 
Lecture Hall, 1999). 

In the fall of 1998, the Masters of Science in Agronomy Distance Education Program at Iowa 
State University (ISU) star^ providing interactive multimedia courses in which most course materials and 
resources were accessed and delivered through WWW and CD-ROM (Iowa State University, 1999). 
One of the goals of this program was to provide a way for professionals woridng in industry and the 
government to gam an advanced degree in Agronomy without having to attend the campus in person. The 
curriculum consists of 12 courses, a l-credh workshop, and a 3-credit creative component, which totals 30 
semester credits (Howard, 1999). The courseware integrates content material on CD-ROM with the 
interactive tools of WebCT (WebCT, 1999) on an ISU server. The WebCT tools allow students to interact 
electronically with dieir instructors and classmates by u tilizing a course calendar, discussion board, chat 
room, student homepages, and grovq> project organization, etc. The program began with an enrollment of 
fifteen students in a pilot program in ie 98/99 academic year. Currently forty-two students are enrolled in 
the program 

At the initial development stage of the Master of Science in Agronomy Distance Education 
Program at ISU, conducting formative evaluations is needed to obtain more understanding and knowledge 
about Web-based leaming/teaching and to improve and strengthen the program The purpose of this paper 
is to report the various formative evaluation activities that have been conducted for the Masters of Science 
in Agronomy Distance Education Program, the results of evaluations, actions taken to improve the courses, 
and to provide useful recommendations for others involved in Web-based course development 
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BESTCOPY AVAILABLE 



Evaluations 



Formative evaluations have been conducted along with the course development of the Masters of 
Science in Agronomy Distance Education Program at ISU. Faculty members and students were the 
primary audience of the evaluations. 



Faculty 

I 

I Agronomy fk;ulty members at Iowa State University were surveyed regarding their perceptions of 

i the MastOT of Science in Agronomy Distance Education Program in the spring of 1 998 (Bom and Miller, 

1999). Findings showed that those faculty members having experiences with distance education held mcro 
j positive perceptions toward Web-based instruction than those not having experiences. Table 1 shows the 

I coini^son of faculty perceptions by whether or not they were involved in other distance courses, 

familiarity of the M.S. in Agronomy Degree Program, whether or not they were involved in the M.S. in 
I Agronomy Degree Program, and whether or not they have viewed a M. S. in Agronomy degree program 

I lessoa Faculty perceptions were significantly positive if the faculty were familiar with the Agronomy 

Distance Education Program, and had viewed a lesson developed by the M. S. in Agronomy degree 
j program. Faculty members were the most positive about the comparability between Web-based and on- 

campus courses in regard to challenge. They also thought the university should develop more Web-based 
courses and integrate them into the curricula (Table 2). 

Furthermore, informal comments regarding the on-line course development were collected fix>m 
the instructors involved in the program. Faculty revealed that developing and managing the on-hne courses 
required them to have a larger amount of time commitment than traditional courses. For example, one 
instructor indicated that to interact with the students, she averaged half an hour per week for one student. 



Table 1 . A comparison of faculty perception of Web-based distance education and 
M. S. in Agronomy degree program by selected variables (n = 36) ® 



Variable 

Whether involved in other distance 
education courses 



Overall Perception of Web- Perception of the MJS. in 
based Distance Education Agronomy Degree Program 
Mean” SD t-value Mean” SD t-^ue 



No 


3.38 


Yes 


3.70 


Whether famihar with the MS. in 
Agronomy degree program 
No 


3.30 


Yes 


3.70 


Whether involved in the MS. in 
Agronomy degree program 
No 


3.36 


Yes 


3.60 


Whether viewed a MS. in Agronomy 
degree program lesson 
No 


3.22 


Yes 


3.83 



“Adapted fix>m Bom and Miller (1999) 

1 = strongly disagree, 2 = disagree, 3 = undecided, 
*2<.05 



0.61 


-1.4 


3.03 


0.55 


-2.47* 


0.66 




3.53 


0.57 




0.61 


-2.15* 


2.93 


0.54 


-3.35* 


0.61 




3.50 


0.52 




0.59 


-123 


2.99 


0.50 


-2.27* 


0.68 




3.34 


0.65 




0.60 


-3.41* 


2.88 


0.46 


-4.52* 


0.49 




3.58 


0.52 





= agree, 5 = strongly agree 
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Table 2. Means and standard deviations for faculty perceptions of Web-based distance education and the 
M.S. in Agronomy Distance Education Degree Program (n = 36)® 



i 



t 



Statement 

OveraO perception of Web-based D£. 

Web-based, distance education courses can be as challenging as on-campus courses. 

Web-based, distance education courses should become an integrated part of university 
curricula. 

Our department needs to develop more Web-based, distance education courses. 

If I were a student, I would consider enrolling in a Web-based, distance education 
course or program. 

Web-based, distance education courses are as academically challenging as on-campus 
courses. 

Web-based, distance education courses should be offered as substitutes for some on- 
campus courses. 

Web-based, distance education courses can not be as effective as cm-campus courses. 

Students spend less time working on Web-based, distance education courses than on- 
campus courses. 

I would consider teaching a Web-based, distance education course. 

Teaching a distance education course would improve my on-campus teaching. 

Effective student-professor interaction is not possible in Web-based, distance 
education courses. 

On-line degrees should not be valued as equivalent to on-campus degrees in the job 
market. 

Perception of the M.S. in Agronomy Degree Program. 

The time and effort expended on the Master of Science in Agronomy Distance 
Education Degree Program is not appropriate. 

A Master of Science in Agronomy Distance Education Degree Program will be 
perceived by employers as having similar status or value as compared to an on- 
campus Master of Science degree. 

The Master of Science in Agronomy Distance Education Degree Program is as 
rigorous as an on-canqjus Master of Science Degree Program. 

® Developed by Bom and Miller (1999) 

1 = strongly disagree, 2 = disag^, 3 = undecided, 4 = agree, 5 = strongly agree 

® Indicates negatively worded items that were reverse coded. 



Mean*’ 


SD 


3.46 


0.63 


4.00 


0.92 


3.98 


0.75 


3.67 


0.93 


3.50 


0.89 


3.43 


0.91 


3.38 


1.03 


3.33' 


1.14 


3.33' 


0.61 


3.31 


1.07 


3.31 


1.05 


3.26' 


0.96 


2.98' 


1.14 


3.15 


0.59 


3.27' 


0.59 


3.10 


0.77 


3.10 


0.94 



Students 

“In addition to seeking for faculty*s opinions regarding the program, students were' also asked to 
evaluate the Web-based courses developed by the program. One pre-pilot focus groiq> study included 
interviews with the students after they previewed the lessons in August of 1998 (Singh, 1999). It was 
found that in general the students liked the lessons and they discovered the lessons were better than what 
they had expected. They thought that they were learning as much in the distance environment as they 
would on campus. The students said that the interactive exercises made the lessons interesting and helped 
them in understanding the concepts. Many of the students also said that more interactive exercises should 
be included in the lessons. They expressed that it would be helpful to have access to the Glossary at all 
times, not just from specific words in the lessons. Most of them were a little confused about their location 
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within a lesson so they suggested that it would be helpful to have a lesson map to the headings and 
subheadings within the lessons. Another suggestion was to have a list of study questions and assignments 
so that the students could be sure that they had not missed any question or assignment. The students liked 
the multimedia presentations and the ease of playing them. They suggested that audio should be added 
Moreover, all the students agreed that the lessons should help them in making the connection between 
theory and its practical application. They also ^predated the opportunity to access a higher education 
degree program and the efforts made in conducting the pilot evaluation for the lesson inqxrovement. 

Students completed a questionnaire regarding their perceptions of Web-based distance education 
and the M.S. in Agronomy Distance Education Degree Program in the fall of 1998. Students agreed that 
the computer they used for the program was sufficient for course materials and they were satisfied with the 
technical supports they received (Table 3). They also believed that they had ^>propriate computer skills 
when they began their work in the M.S. in Agronomy program. They thought that the M.S. degree in 
Agronomy Distance Education Program should be of equal value to a traditional on-can^us degree. They 
agreed that they would not be able to work on their Masters if it were not for this distance degree in 
Agronomy. They also agreed that they preferred distance courses to on-campus courses. 

Students taking the courses offered in the fell of 1998 and spring of 1999 completed a survey 
qu^onnaire regarding various aspects of the courses. Overall, students agreed that the courses stimulated 
their interest in the subject matter. They thought their understanding of the subject has increased because 
of their e?q)enences in taking the courses. They thought that the images and animations helped them in 
imderstoding and visualizmg the lesson content and the course content was relevant to their career goals. 
In addition, they indicated that audio and video should be iiKorporated more often in explaining the 
concepts. They did not think that the navigation through the lessons was clear and logical or there was 
adequate explanation for all the acronyms and Greek symbols used within the lessons. 

In the fell of 1999, one student in each course was asked to evaluate individual lesson weekly. 
Three questions were asked in the weekly lesson evaluation. The first question was: “What were the most 
effective parts of the lesson for you?” Up to the date of writing this p^ier, most of the students responded 
that tile most effective part of the lessons were the multimedia presentations of concepts. They also 
repor^ that the lessons effectively used m^s, charts, and tables to back up the text and the lesson 
materials were well organized. Students liked the use of features such as FYI, In Detail, Study Questions, 
and Glossary to siqjplement information presented while maintaining the flow of material. The hands on 
activities went with these interactive features **aie the best learning” according to one student. The second 
que^on was: “W^t did you like least about the content and/or design of this lesson?” Some students 
replied they did not like those lessons wiiich were extremely too long for a week. One student revealed 
that he did not like the activities that required him to be on-line because he needed to be able to be in a 
motel, at home, or at some other site to study whenever possible. One student was fiustrated to work on the 
symbols of the math or equations fi-om MS Word for the assignment. Technical problems are always not 
patientiy welconw. However, only one student reported having technical problems. The problem was that 
the animations did not woric properly even though he had downloaded all the plug-ins required. One 
student had probleiM with exams. He said that he would like to know what is ejq>ected and what concepts 
were deemed to be in^xjrtairt for fee exams. Regarding what could be added to the lessons to enhance 
student learning, students would like to see more multimedia presentations to help convey conceptual 
topics more effectively and more real pictures as real world examples. One student suggested to add an 
appendix with a review, problems, or recommended siqjplemental text that will be important to study the 
sul^ect matter. Another student would like to see some case studies where the modeling concepts were 
^lied. 



Actions of Improvement 

Based on the results of these evaluations, improvements have been made. For example, lesson 
maps, examples of the calculations, and glossaries have been added to the courses. More multimedia 
presentations have been developed Navigation has been inqiroved by adding back to the previous lesson 



and to the immediate following lesson. A search function for topics and words within each course has also 
been tested 



Table 3. Means and standard deviations for student perceptions of Web-based distance education and the 
M.S. in Agronomy Distance Education Degree Program (n = 1 1)® 



Statement 


Mean*^ 


SD 


Logistics 


3.15 


0.86 


The computer I used for the program was sufficient for course materials. 


3.64 


0.50 


Questions on technical problems were answered quickly and efficiently by the MS in 


3.55 


0.69 


Agronomy staff. 

I felt comfortable with the computer systems and software. 


3.27 


1.19 


The instructions that were provided to help set-iq) my computer for the program were 


3.09 


0.94 


effective. 






I was given an adequate amount of time to become comfortable with the technology 


3.00 


1.18 


before I was required to participate. 

The number of on-campus meetings was adequate. 


2.91 


1.14 


Correspondence was answered by the instructor within two business days of receipt 


2.73 


1.49 


The library was available for research. 


2.30 


0.48 


Research and reference materials were accessible. 


2.30 


0.48 


If this course had not been offered via distance learning, I would have commuted to 


.33' 


0.50 


ISU to take a similar course. 






Computer SldDs 


2.49 


0.89 


I had the appropriate computer skills when I began my work in the MS in Agronomy 


3.18 


1.25 


program. 


2.91 


0.54 


Students should be informed about the level of computer skills necessary for the 


classes prior to enrolling in the program 
As a result of my work in the program, my computer skills have improved 


2.55 


0.69 


I needed more experience working with Excel for my work in the program. 


2.45 


0.93 


I needed more e?q)erience working with the Internet for my work in the program 


1.36 


1.03 


Time and Convenience 


2.70 


0.98 


This degree should be of equal value to a traditional on-campus degree. 


3.45 


0.93 


I would not be able to work on my Masters if it were not for this distance degree. 


3.27 


0.90 


1 prefer distance courses to on-campus courses. 


3.09 


0.94 


I never felt that the technical problems of diis course were overvdielming. 


2.36 


1.12 


The classes should adhere to the traditional semester scheduling. 


2.36 


1.12 


I spent more time per week on the distance classes than I would have in a traditional 


2.27 


0.79 


graduate course taught on campus. 

The amount of time spent on each class was consistent fiom week to week. 


2.09 


1.04 



“Developed by Singh (1999) 

1 = strongly disagree, 2 = disagree, 3 = undecided, 4 = agree, 5 = strongly agree 
° Indicates negatively worded items that were reverse coded 
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Conclusions and Recommendations 

Based on the results of these on-going formative evaluations, the following conclusions were 

drawn: 

• Those faculty members having experiences with distance education and the M. S. in 
Agronomy degree program held more positive perceptions toward Web-based instructioa 

• Faculty recognized that more time commitment was needed to develop and manage the Web- 
based courses than traditional ones. 

• Students were positive about interactive multimedia activities. 

• Students wanted clear structure for navigation in each web page. 

• Students believed they were learning as much in the distant environment as they would on 
campus. 

• Students appreciated the access to a higher education degree program provided. 

• Students also appreciated the efforts made in conducting the evaluations for the in^>rovement 
of the lessons. 

Recormnendations were made for educators who are int^ested in Web-based course development 
as follows: 



• Oppor^ties should be provided for faculty members to be involved with Web-based 
education to ensure the acceptance and adoption of the distance program. 

• More interactive multimedia activities should be included in the Web-based lessons. 

• D^elopers should make navigation of the lesson pages as simple and clean as possible. 

• Distance education degree programs should be pro\aded 

• Qn-going formative evaluations are needed to obtain evidence to help with the design of foe 
Web-based courses and continue to improve foe quality of foe courses. 
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